Google Bigtable

Please comment one line (fmt.Fprintf(w, “Search received: %s %s %s”, lat, lon, ran)) as IOS does not like to get string. Make sure the response of \post is empty

Bigtable

BigTable is one of the first NoSQL solutions in the world. It’s the same database that powers many core Google services, including Search, Analytics, Maps, and Gmail.

Apache HBase is the open source version of BigTable.

By definition, A Bigtable is a sparse, distributed, persistent multidimensional sorted map. (In my word, it’s a key­value store.) The map is indexed by a row key, column key, and a timestamp; each value in the map is an uninterpreted array of bytes.

Main features that you must know about BigTable

• Map
• Persistent: BigTable vs. Memcache
• Distributed (GFS/HDFS, etc.): data is replicated across a number of participating nodes vs. MySQL
• Multidimensional: row, column (column family), timestamp. You may scan based on their ranges.
• Sparse: A given row can have any number of columns in each column family, or none at
  all. The other type of sparseness is row­based gaps, which merely means that there may be gaps between keys.

BigTable vs. ElasticSearch vs. Datastore vs….

• BigTable is persistent storage (ES is not persistent, may lose data)
• ElasticSearch is search engine with complicated query support and better read
  performance
• BigQuery is for offline analysis not for serving user traffic (scale is small)
• MongoDB is NoSQL.
• BigTable vs. ElasticSearch vs. MongoDB vs. BigQuery vs. Dataflow
  • BigTable = NoSQL + Cloud
  • MongoDB = NoSQL
  • ElasticSearch = NoSQL + Query Optimization
  • BigQuery = MySQL­like + Cloud
  • Dataflow = MapReduce + Cloud

Interview question:

Why you need BigTable (or BigQuery) while you already have ElasticSearch (BigTable/BigQuery)

More readings

• http://chouqin.github.io/blog/2013/10/24/bigtable/
• http://blog.csdn.net/opennaive/article/details/7532589
• https://www.quora.com/What­problems­did­Spanner­solve­that­BigTable­fall­short­of

Bigtable has a concept of column family and column. Column family is a set of columns.

Question: Why column family?
Answer: allow better management of data such as versioning strategy.

Create BigTable in Google Cloud

Click this link https://cloud.google.com/bigtable/docs/quickstart­cbt and then click ‘ENABLE THE APIS’

To Enable an API.

Open ‘BigTable’ in Google Cloud,

Create an instance called ‘around­post’. Change instance type to development.

Set the type to ssd and instance type to development

Enter gcloud in your terminal

Install cbt in your cloud terminal (not your local terminal). When asked ‘Do you want to continue?’ Enter y. Proceed without waiting it to finish.

sudo gcloud components update sudo gcloud components install cbt
```     

To ensure cbt is installed, enter `which cbt` in your cloud terminal If you don’t see it, you cbt is not installed, please do the install again.

/google/google-cloud-sdk/bin/cbt

In your cloud terminal, enter

echo project = YOUR_PROJECT_ID > ~/.cbtrc echo instance = around­post >> ~/.cbtrc
cbt createtable post
cbt createfamily post post
cbt createfamily post location

If you see such results, the bigtable column and column family are both created (This step is a MUST for next lessons). Please check it carefully.

![6. Bigtable-2019-5-3-10-58-1.png](https://raw.githubusercontent.com/Luorinz/images/master/6.%20Bigtable-2019-5-3-10-58-1.png)


## Write data into BigTable
### Open your local terminal and cloud terminal, enter

go get cloud.google.com/go/bigtable

Instructions on writing data to BigTable
+ https://cloud.google.com/bigtable/docs/go/reference


### Update your code in handlerPost and saveToBigTable. Replace project name with your project.
```Go
func saveToBigTable(p *Post, id string) {
	ctx := context.Background()
	// you must update project name here
	bt_client, err := bigtable.NewClient(ctx, PROJECT_ID, BT_INSTANCE)
	if err != nil {
		panic(err)
		return
	}

	tbl := bt_client.Open("post")
	mut := bigtable.NewMutation()
	t := bigtable.Now()
	
	mut.Set("post", "user", t, []byte(p.User))
	mut.Set("post", "message", t, []byte(p.Message))
	mut.Set("location", "lat", t, []byte(strconv.FormatFloat(p.Location.Lat, 'f', -1, 64)))
	mut.Set("location", "lon", t, []byte(strconv.FormatFloat(p.Location.Lon, 'f', -1, 64)))

	err = tbl.Apply(ctx, id, mut)
	if err != nil {
		panic(err)
		return
	}
	fmt.Printf("Post is saved to Big Table : %s\n", p.Message)

}

Based on what Google suggest, how to save our Post into BT?

tbl := client.Open("mytable")
mut := bigtable.NewMutation()

mut.Set("links", "maps.google.com", bigtable.Now(), []byte("1")) 

mut.Set("links", "golang.org", bigtable.Now(), []byte("1"))

err := tbl.Apply(ctx, "com.google.cloud", mut)

Uncomment the two lines in const and replace the PROJECT_ID with your own project id.

const (
INDEX = "around"
TYPE = "post"
DISTANCE = "200km"
// Needs to update with your real project id PROJECT_ID = "xxxx"
BT_INSTANCE = "around­post"
// Needs to update this URL if you deploy it to cloud. ES_URL = "http://AWS_ADDRESS:9200"
)

Add two imports

package main

import (
elastic "gopkg.in/olivere/elastic.v3" "fmt"
"net/http"
"encoding/json"
"log"
"strconv"
"reflect"
"context" "cloud.google.com/go/bigtable" "github.com/pborman/uuid"
)

Try to build it locally to see if there are any errors.

go build *.go

Update your codes in cloud terminal (Either copy or github) Offer share

After you copy or update codes into cloud terminal, do a deployment again.

Open Postman, test post and search

Read data from BigTable

In the cloud terminal, enter

cbt read post

You should be able to find all the entries stored in BT.

How to debug? Check Logging­>logs.

Note that: if you close your cloud terminal, you may or may not need to install cbt again as cloud terminal is actually a vm.

Things you need to know

1. setup bigtable
2. BigTable vs. ElasticSearch vs. MongoDB vs. BigQuery vs. Dataflow
   • BigTable = NoSQL + Cloud
   • MongoDB = NoSQL
   • ElasticSearch = NoSQL + Query Optimization
   • BigQuery = MySQL + Cloud
   • Dataflow = MapReduce + Cloud